

COMPOSING A REALIGNED IMAGE 



ABSTRACT OF THE INVENTION 



This invention provides methods, apparatus and article 
of manufacture used as a countermeasure to image 
distorting. The present invention involves 
automatically detecting the presence of distortion in a 
presumed distorted image, measuring the magnitude and 
type of distortion, and finally creating a realigned 
image. Once image distortion is removed, conventional 
invisible watermark extraction methods are employed to 
extract the watermark from the realigned image. The 
automatic method does not depend on a process of 
visually examining a composite image and recording the 
coordinates of pixel locations closest to common image 
features in a distorted image and reference image. 
Generally, the presumed distorted image is resized to 
the same size as the reference image. Reference 
centers at at least three distinct points that do not 
form a straight line are selected. At each reference 
center, a sub-image is excised. Pairs of sub-images, 
one excised from the reference image and the other 
excised from the distorted image, having the same 
reference center are used in the process of removal of 
the distortion. 
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1 FIELD OF THE INVENTION 

2 This application relates to the field of digitized 

3 imaging. It is more specifically directed to 

4 determining and removal of image distortion from an 

5 image . 



6 BACKGROUND OF THE INVENTION 

7 With the development of means of production and 

8 circulation of digital images, and the means of 

9 imbedding relatively invisible watermarks into digital 

10 images ostensibly to convey ownership of the image, 

11 there is now financial incentive to attack an imbedded 

12 watermark in an attempt to render it non-extractable. 

13 Pixel locations of a watermarked image are presumed to 

14 correspond to those in an unmarked original image . 

15 Generally, the watermark is imbedded by altering only 

16 the values of the pixel components of the original 

17 image, not their geometric positions. This may be 

18 . accomplished employing such methods as described in US 

19 Patent 5,825,892 which is incorporated herein by 

20 reference in its entirety. 

21 Some methods of attacking an imbedded watermark rely on 

22 constructing a new image that is a geometrically 
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1 distorted copy of the watermarked image. This new 

2 image is herein referred to as a distorted copy. 

3 Pixels in the distorted copy are placed at subtly 

4 distorted positions relative to those in the 

5 watermarked image. Pixel component values in the 

6 distorted copy are determined by two-dimensional 

7 interpolation of component values of enclosing pixel in 

8 the watermarked image. No constraints can be placed on 

9 the types of pixel position distortion an attacker 

10 might choose to use. To those skilled in the art, 

11 however, it is obvious that excessive pixel 

12 position-distortion will cause the distorted copy to be 
O 13 a caricature of the watermarked image, thus diminishing 

14 or destroying its economic value. Whether a distortion 

\R 15 is excessive is a subjective measure. For a distorted 

: .. 

jfcfe 16 copy to be useful, it requires that linear or nonlinear 

*^ 17 distortion methods that are used by an attacker have to 

s 18 be used sparingly and in such a manner as to produce 

Jtntc 

Cj 19 smoothly varying and relatively small position 

W 20 distortions. This is so as to be essentially 

ijy 21 unobjectionable and casually unnoticeable to untrained 

^ 22 observers. The human visual system, as a qualitative 

23 measuring device, can be relied upon to readily detect 

24 excessive distortion. It is desirable to have a method 

25 of defense that requires little or no limits to be 

26 placed on pixel position-distortions produced by the 

27 attacking method. 
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SUMMARY OF THE INVENTION 
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1 Another aspect of the present invention enables a 

2 digital image watermark to be extracted from a 

3 geometrically distorted copy of a reference image. 

4 Other aspects and a better understanding of the 

5 invention may be realized by referring to the Detailed 

6 Description. 

7 BRIEF DESCRIPTION OF THE DRAWING 

8 These and other objects, features, and advantages of 

9 the present invention will become apparent upon further 
10 consideration of the following detailed description of 

j~ 11 the invention when read in conjunction with the drawing 

!^ 12 figure. 

lo 

m 

Q 13 DESCRIPTION OF THE INVENTION 

*S 14 The present invention provides a method, apparatus and 

15 article of manufacture by which the distortion in a 

16 distorted copy of an image is automatically measured 

17 and removed sufficiently well that a relatively 

18 invisible image watermark extraction is possible. In 

19 an embodiment of the invention, the method employs a 

20 reference image relative to which measurements of 

21 distortion are made. For example, an original unmarked 

22 image or a relatively invisibly watermarked copy of the 
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1 original image, called a watermarked image, serve 

2 equally well as the reference image. 

3 The automatic method composes a realigned image in a 

4 way that does not depend on a process of visually 

5 examining a composite image and recording the 

6 coordinates of pixel locations closest to common image 

7 features in a distorted copy and reference image, as is 

8 the case in the semiautomatic method of Application 

9 Number 09/240,242, cross-referenced herewith. 

10 As with the semiautomatic method, if the presumed 

O 11 distorted copy is not the same size as the reference 

^ 12 image, it is made so by shrinking or enlarging the 

m 13 distorted copy using a method such as pixel 

\& 14 interpolation or pixel extrapolation. The distorted 

% % 15 copy, after being geometrically resized, is herein 

16 referred to as a geometric alteration. There are well 

Q 17 known automatic and manual methods for resizing images. 

kQ 18 At the next step, the automatic process differs from 

"** 19 the semiautomatic process. At least three common pixel 

20 locations in the two images, herein referred to as 

21 reference centers, not lying in any straight line are 

22 required for the automatic method of this invention. 

23 in an example embodiment, a coarse but regularly spaced 

24 simulated mesh is superimposed identically on each of 

25 the two images. Each knot in the mesh forms one of the 

26 common pixel locations. The integer coordinates of the 

27 pixels nearest each of the at least three common pixel 
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1 locations are the reference centers. At each reference 

2 center, a segment of each image, herein referred to as 

3 a sub -image, is excised, thus producing for each image 

4 as many sub-images as there are common pixel locations. 

5 The horizontal and vertical dimensions of the 

6 sub -images, in integer pixel coordinates, are based on 

7 the common horizontal and vertical dimensions, 

8 respectively, of the images. The pair of sub -images, 

9 one excised from the reference image and the other 

10 excised from the geometric alteration, having the same 

11 reference center are herein referred to as 

12 corresponding sub -images. A typical embodiment employs 
f% 13 three common pixel locations. 

w 

IM 14 When the geometric alteration is laid upon the 

15 reference image, the intersection of pixels in the 

^ 16 reference image with pixels in the geometric alteration 

s; 17 is an area of each image called the safe area. 

18 Although initially the safe area is the entire area of 

O 19 the geometric alteration and the reference image, on 

,p 20 subsequent iterations of this realignment method the 

^ 21 safe area may be smaller than the reference image. 

22 Each sub -image is centered at its respective reference 

23 center. If any pixel of a sub -image extends beyond the 

24 boundaries of the safe area, such as at the edges of 

25 the safe area, then the top or bottom edge of the 

26 corresponding sub- images, and the left or right edge of 

27 the corresponding sub -images, if also necessary, are 

28 repositioned. The two subimages, in unison, are 

29 repositioned vertically and horizontally by the minimum 



7 

DOCKET NUMBER: Y0999-Q34 



number of pixel locations necessary so that no pixel of 
either sub -image lies in whole or in part outside of 
the safe area. The corresponding reference center is 
adjusted to lie at the center of the repositioned 
sub -images. Reference centers, after this adjustment 
process, are referred to as adjusted centers, whether 
they were adjusted or not. Note that in the case where 
the sub -image horizontal width, J, and vertical height, 
J , are even integers, such as powers of two, and the 
coordinates of the sub -images are indexed 0 to M and 
0 to J'l, respectively, the integer coordinates of the 
center of each sub -image are defined to be 1/2 and J/2. 
Also note that J should be less than the width and J 
should be less than the height of the reference image 
and the geometric alteration. 

The n-th pair of sub -images, one from the reference 
image designated the n-th reference sub- image and a 
corresponding one from the geometric alteration 
designated the n-th distorted sub- image, is used to 
compute a two-dimensional cross -correlation surface 
relating the distorted sub- image with the reference 
sub -image image. Those skilled in the art will 
recognize that there are many methods that may be used 
to compute or approximate a cross -correlation surface 
relating the two corresponding sub -images. Regardless 
of the method used, any method that produces the 
intended result of determining the horizontal and 
vertical offsets of the distorted sub -image relative to 
the reference sub -image that achieves a good match may 
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1 be employed . For example, a good match has offsets 

2 that can not be improved by more than the spacing of 

3 1.5 pixels in any direction by using any other means of 

4 pattern matching. A better match is readily achievable 

5 if the distortions are generally linear. Watermark 

6 detection seldom requires a match better than a 0.5-2.0 

7 pixel spacing. If required, more complex iteration 

8 techniques may be used to achieve a best match. 

9 In an example embodiment, forward and inverse discrete 

10 Fast Fourier Transforms (FFT's) are used to compute a 

11 cross -correlation surface. Note that for methods using 
0 12 FFT's, all sub-images are converted to monochrome, if 

fi 13 not already so. The non- integer interpolated 

fli 14 horizontal and vertical offsets of the peak of the 

j iA 15 cross -correlation surface, p n and g n , relative to the 

^ 16 origin of the cross -correlation surface are used as 

s . 17 additive offsets relating the center of the n-th 

•!1 18 distorted sub-image relative to the n-th corresponding 

d 19 adjusted center. Thus, for each pair of sub-images, 

tig 20 the coordinates of the adjusted center are the center 

,|y 21 on the n-th reference sub -image. The sums of the 

22 coordinates of the adjusted center plus the offsets of 

23 the peak of the n-th cross -correlation surface become 

24 the approximate coordinates of the center of the 

25 distorted sub image, u n = x n +p n and v n = y n +qn- The 

26 coordinates of the offset center of the distorted 

27 sub-sub image and the adjusted center, herein referred 

28 to as computed pixel coordinates, are analogous to the 

29 measured pixel coordinates of common features visually 
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1 selected from the geometric alteration and reference 

2 image of the previously cited semiautomatic method. 

3 Using the computed pixel coordinates instead of the 

4 measured pixel coordinates, the coefficients matrix, A, 

5 of the pixel position interpolation equations are 

6 computed, as in the semiautomatic method. 

7 In another example embodiment, forward and inverse 

8 discrete Fast Fourier Transforms (FFT's) are used to 

9 compute a modified cross -correlation surface. The 

10 magnitudes of the Fourier transform coefficients are 

11 modified to make said magnitudes uniform. An inverse 

12 Fourier transform is used to compute the modified 
;0 13 cross -correlation surface. 

14 In yet another example embodiment, a weighted sum of 

i 

d|j 15 the ordinary and modified Fourier transform 

^ 16 coefficients is formed before using an inverse Fourier 

W 17 transform to compute a weighted cross correlation 

"frj 

d 18 surface. 

*B 19 In an embodiment of the automatic method, use of a 

20 statistical screen is incorporated into the next step. 

21 Use of a statistical screen is generally not needed in 

22 the semiautomatic method. Because the selection of 

23 sub -images is done indiscriminately, it is possible 

24 that some sub -images may have few features (or even 

25 none) . The cross -correlation surfaces computed from 

26 such sub -images may be relatively flat and have a 

27 misleading peak. An additional test is used to 
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discriminate against statistical out -flyers that can 
occur from such misleading peaks. As used herein, an 
'out -flyer' is a value from a set of values that 
deviates so greatly from the other values in the set 
that it is statistically unlikely to be a member of the 
set. 

An example statistical screen discriminating against 
out -flyers is embodied as follows. Each of the pairs 
of computed pixel coordinates, u n and v u , is processed 
by the interpolation equations to give a proposed 
reference center, x n ' and y n ' . The Euclidean distance 
between the proposed reference center and the 
corresponding adjusted center, x n and g n , is computed 
for each of the n sub -images. Out -flyers are deleted 
from the set of n Euclidean distances, largest first, 
based on their value being above a first threshold 
value. If any out-flyer is deleted, the offset center 
of its corresponding distorted sub -image and adjusted 
center are also deleted. The coefficients, A, are 
recomputed using the undeleted pairs of pixel 
coordinated, but never with fewer than three pairs of 
pixel coordinates. 

From this point on, the automatic and semiautomatic 
methods generally parallel one another. The square 
submatrix, S A , of four of the coefficients of the 
interpolation equations is factored into four primitive 
image manipulation matrices. The geometric alteration 
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is manipulated by the four primitive image 
manipulations to form the reoriented image. 

The entire restoration process is advantageously 
repeated iteratively by substituting the previously 
reoriented image for the geometric alteration before 
each subsequent iteration. For the automatic method, 
it has been found that for images with significant 
distortion, as many as three iterations are warranted. 
For attacks with more nonlinear distortion, additional 
iterations may be used to further improve reorientation 
of the geometric alteration. The iteration process is 
terminated when examination of the Euclidean distances 
shows that the reduction of all Euclidean distances 
relative to those from the previous iteration is less 
than a second threshold. Finally, the reoriented image 
is realigned left or right and up or down relative to 
the reference image, based on the offset coefficients c 
and f from the interpolation equations, to form the 
realigned image. 

An overview of the steps of an example automatic 
realignment of a presumed distorted copy is shown in 
the figure. First, if the presumed distorted copy is 
not the same size as the reference image, it is shrunk 
or enlarged to make it the same size using any image 
resizing method; the resized distorted copy is herein 
referred to as the geometric alteration, (102) . 
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In a particular embodiment, a common coarse but 
regularly spaced mesh having n knots is superimposed on 
both of the images, (104) . The knots of the mesh serve 
as reference points for each of the images. A list of 
the horizontal and vertical coordinates, x n and g nr of 
pixels lying closest to the knots of the mesh is built; 
these coordinate pairs are herein referred to as the 
reference centers, (106) . A sub -image, having 
dimensions I and J and centered at each of the 
reference centers, is excised from each of the images 
and converted to monochrome, if not already so, (108) , 
producing n corresponding pairs of sub images. A safe 
area is determined as the intersection of pixels common 
to the geometric alteration and reference image (110) . 
If any part of any sub -image lies beyond the boundaries 
of the safe area, it is repositioned left or right and 
up or down a minimum number of pixel locations until 
both it and its corresponding sub -image lie within the 
boundaries of the safe area, and their common reference 
center is adjusted to again lie at the center of the 
repositioned sub-images, (112) . A two-dimensional 
cross correlation surface is computed from each of the 
n pairs of sub -images, (114) . The non- integer 
horizontal and vertical coordinate offsets, p n and g nf 
of the greatest peak on each of the n cross -correlation 
surfaces are determined by two-dimensional 
interpolation, (116) . The distorted reference centers 
are computed by adding the coordinate offsets to their 
corresponding adjusted centers, (118) . Using the 
adjusted centers and the distorted reference centers in 
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